Controlling communication between a local device and a remote terminal device

ABSTRACT

There is provided an information processing device, including: a camera that captures a real space; a communication unit that communicates with a terminal device used by a remote user; a streaming control unit that streams a first video captured by the camera from the communication unit to the terminal device if the information processing device itself is selected by the remote user from among a plurality of devices that capture the real space; and a display that displays a user image of the remote user while the first video is being streamed to the terminal device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2015/063553 filed on May 12, 2015, which claimspriority benefit of Japanese Patent Application No. JP 2014-123564 filedin the Japan Patent Office on Jun. 16, 2014. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a terminal device.

BACKGROUND ART

Recently, communication tools for remote communication are beingutilized in a variety of fields, such as business, education, andamusement. One representative example of such communication tools is avideoconferencing system (for example, see Patent Literature 1 below).In communication through a videoconferencing system, a local userpresent in a local environment in which the system is installed and aremote user present in a remote location participate. Additionally, thesystem streams a video of the local environment to the remote user'sterminal device. Other examples of communication tools for remotecommunication are videophones and video chat.

CITATION LIST Patent Literature

Patent Literature 1: JP 2008-306327A

SUMMARY OF INVENTION Technical Problem

With existing communication tools, in many cases, the camera in thelocal environment is set by the local user, and is installedsemi-permanently. If the remote user is approved to participate in thecommunication, the remote user is able to view a video of the localenvironment. However, the remote user is unable to move around in thelocal environment, and both the remote user's field of view and actionsin the local environment are constrained greatly by the setup of thesystem.

Accordingly, an objective of technology according to the presentdisclosure is to reduce inconvenience for the remote user discussedabove during remote communication, and realize rich remote communicationwith a higher degree of freedom.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing device, including: a camera that captures a real space; acommunication unit that communicates with a terminal device used by aremote user; a streaming control unit that streams a first videocaptured by the camera from the communication unit to the terminaldevice if the information processing device itself is selected by theremote user from among a plurality of devices that capture the realspace; and a display that displays a user image of the remote user whilethe first video is being streamed to the terminal device.

According to the present disclosure, there is provided an informationprocessing method executed by an information processing device providedwith a camera that captures a real space and a communication unit thatcommunicates with a terminal device used by a remote user, theinformation processing method including: streaming a first videocaptured by the camera from the communication unit to the terminaldevice if the information processing device itself is selected by theremote user from among a plurality of devices that capture the realspace; and displaying a user image of the remote user on a display whilethe first video is being streamed to the terminal device.

According to the present disclosure, there is provided a terminaldevice, including: a communication unit that communicates with aplurality of devices that respectively capture a real space; a streamingcontrol unit that plays back a first video depicting the real spacestreamed from a device selected by a remote user using the terminaldevice from among the plurality of devices; and a device control unitthat causes a display of the selected device to display a user image ofthe remote user while the first video is being played back by thestreaming control unit.

According to the present disclosure, there is provided an informationprocessing method executed by a terminal device provided with acommunication unit that communicates with a plurality of devices thatrespectively capture a real space, the information processing methodincluding: playing back a first video depicting the real space streamedfrom a device selected by a remote user using the terminal device fromamong the plurality of devices; and causing a display of the selecteddevice to display a user image of the remote user while the first videois being played back.

Advantageous Effects of Invention

According to technology according to the present disclosure,inconvenience is reduced for the remote user during remotecommunication, and rich remote communication with a higher degree offreedom is realized. Note that the effects described above are notnecessarily limitative. With or in the place of the above effects, theremay be achieved any one of the effects described in this specificationor other effects that may be grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram for describing an overview of aninformation processing system according to an embodiment.

FIG. 2 is an explanatory diagram for describing an example of a localenvironment of the information processing system illustrated as anexample in FIG. 1.

FIG. 3A is a block diagram illustrating an example of a configuration ofa local device configured as a stationary device.

FIG. 3B is a block diagram illustrating an example of a configuration ofa local device configured as a portable device.

FIG. 3C is a block diagram illustrating an example of a configuration ofa local device configured as a mobile device.

FIG. 4A is an explanatory diagram illustrating an additional example ofa portable device.

FIG. 4B is an explanatory diagram illustrating an additional example ofa mobile device.

FIG. 4C is an explanatory diagram illustrating an additional example ofa mobile device.

FIG. 4D is an explanatory diagram illustrating an additional example ofa mobile device.

FIG. 5 is an explanatory diagram illustrating how a video depicting alocal environment appears on a local device.

FIG. 6 is an explanatory diagram illustrating an example of displayimage transitions in a local device when a remote user transfers to anearby device.

FIG. 7 is an explanatory diagram illustrating an example of displayimage transitions in a local device when a remote user transfers from anearby device.

FIG. 8 is an explanatory diagram illustrating an example of a displayimage on a device that takes a picture of two devices when a remote usertransfers between the two devices.

FIG. 9 is an explanatory diagram illustrating an example of a displayimage on a device in which multiple remote users are residing.

FIG. 10 is a block diagram illustrating an example of a configuration ofa remote terminal.

FIG. 11 is an explanatory diagram illustrating an example of a GUI thatmay be displayed on a remote terminal.

FIG. 12 is a block diagram illustrating an example of a configuration ofa management server.

FIG. 13 is an explanatory diagram illustrating an example of a structureof local device data managed on a management server.

FIG. 14 is an explanatory diagram illustrating an example of a structureof remote user data managed on a management server.

FIG. 15 is a sequence diagram illustrating an example of a flow ofinformation processing during login of a remote user into a system.

FIG. 16 is a sequence diagram illustrating an example of a flow ofinformation processing when a remote user transfers from a local deviceto a nearby device.

DESCRIPTION OF EMBODIMENT(S)

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

Also, the description will proceed in the following order.

1. Overview of system

-   -   1-1. Overview    -   1-2. Local environment

2. Example configuration of devices

-   -   2-1. Local device    -   2-2. Remote terminal    -   2-3. Management server

3. Process flows

-   -   3-1. Login to system    -   3-2. Transfer between devices

4. Conclusion

<1. Overview Of System>

[1-1. Overview]

FIG. 1 is an explanatory diagram for describing an overview of aninformation processing system 1 according to an embodiment. Referring toFIG. 1, the information processing system 1 includes local devices 100a, 100 b, 100 c, 100 d, 100 e, and 100 f present in a local environment10, a network 20, a remote terminal 200, and a management server 300.

The local environment 10 may be an arbitrary environment in a real spacewhere real objects may exist. Communication between users in theinformation processing system 1 is conducted in the local environment10. The local environment 10 may be installed in a variety of locations,such as an office, a laboratory, a conference hall, a school, a shoppingmall, an amusement park, or a home, for example.

Each of the local devices 100 a, 100 b, 100 c, 100 d, 100 e, and 100 fis an information processing device existing in the local environment10. The local device 100 a is an electronic chalkboard. The local device100 b is a desktop personal computer (PC). The local device 100 c is asmartphone. The local device 100 d is a tablet PC. The local device 100e is a quadcopter. The local device 100 f is a talking head. Note thatin the following description of this specification, when the localdevices 100 a, 100 b, 100 c, 100 d, 100 e, and 100 f are not beingdistinguished from each other, these devices will be collectivelyreferred to as the local device 100 by omitting the trailing letters inthe reference signs. This applies similarly to the other structuralelements.

Each local device 100 is equipped at least with a camera and acommunication interface. Each local device 100 includes a function ofcapturing a video of a real space using the camera, and streaming thecaptured video of the real space. Furthermore, each local device 100 mayalso stream audio picked up through a microphone. Streaming data fromeach local device 100 may be delivered to the remote terminal 200directly, or relayed to the remote terminal 200 via the managementserver 300. FIG. 1 illustrates six local devices 100, but theinformation processing system 1 may include fewer or more local devices100 of arbitrary types.

The network 20 interconnects the local device 100, the remote terminal200, and the management server 300. The network 20 may be a network ofany type, such as the Internet, a corporate intranet, a home network, ora personal network. The network 20 may be a wired network, a wirelessnetwork, or a combination of the two. The network 20 may also includeone or more pieces of network equipment (not illustrated), such as awireless access point, a router, a switch, a repeater, or a bridge.

The remote terminal 200 is a terminal device used by the remote user.The remote terminal 200 receives video data (and audio data) streamedover the network 20 from any local device 100 existing in the localenvironment 10, and plays back a video (and audio) of the real space inthe local environment 10. In the present embodiment, the local device100 acting as the streaming source is designated the streaming sourcedevice. The streaming source device typically is selected by the remoteuser. Although FIG. 1 illustrates only one remote terminal 200, theinformation processing system 1 may also include more remote terminals200.

The management server 300 is an information processing device thatmanages streaming between the local device 100 and the remote terminal200 in the information processing system 1. The management server 300includes a database that stores device data for each of the localdevices 100 (hereinafter designated local device data), for example. Thelocal device data may be provided to the remote terminal 200 to supportthe selection of a device by the remote user. Additionally, themanagement server 300 may also include a database that stores user dataabout the remote user (hereinafter designated remote user data). Notethat the management server 300 may also be omitted from theconfiguration of the information processing system 1. In this case, thelocal device data and the remote user data may be managed in acentralized manner by one of the local devices 100 or the remoteterminal 200, or managed in a distributed manner by the respectivedevices.

When an existing communication tool like a videoconferencing system isused, typically, a camera and a display are installed in the localenvironment. The setup of the camera and the display may be performed bya local user. The setup of these devices may allow for dynamic controlsuch as pan, tilt, and zoom, for example, but ordinarily, the setup issemi-permanent. The remote user is unable to move around in the localenvironment. Consequently, the remote user's field of view and actionsin the local environment are constrained greatly by the setup of thesedevices.

In contrast, in the present embodiment, a plurality of the local device100 discussed above is installed in the local environment 10. The remoteuser is able to access any freely selectable local device 100 via theremote terminal 200, and view the video from the selected local device100, or in other words, the streaming source device. The selection ofthe streaming source device may be changed even in the middle ofcommunication. On the display of the selected streaming source device, auser image of the remote user (such as a photograph of the user or acharacter image enabling the remote user to be identified, for example)is displayed. For this reason, by viewing the user image displayed bythe local device 100, the local user is able to recognize which remoteuser is residing in which device (that is, which device from which theremote user is viewing a video). The user image may also be a movingimage. In one working example, the local user is also notified of achange in the selection of the streaming source device by the remoteuser through a symbolic animation displayed on the display.

[1-2. Local Environment]

FIG. 2 is an explanatory diagram for describing an example of the localenvironment 10 of the information processing system 1 illustrated as anexample in FIG. 1. Referring to FIG. 2, a local device 100 a isinstalled in the local environment 10. The local device 100 a includes acamera 101 a and a display 105 a. Local users LU11 and LU12 are gatheredin front of the local device 100 a. Also, a user image of a remote userRU11 is being displayed on the display 105 a of the local device 100 a.The remote user RU11 is able to converse with the local users LU11 andLU12 through a microphone and a speaker (not illustrated) of the localdevice 100 a.

Local users LU13 and LU14 are gathered in front of a local device 100 b.The local user LU13 is holding a local device 100 c. A local device 100f is moving in front of the local device 100 b. A user image of a remoteuser RU12 is being displayed on a display of the local device 100 f. Auser image of a remote user RU13 is being displayed on a display of thelocal device 100 c. The remote users RU12 and RU13 are able to conversewith the local users LU13 and LU14 through the local devices 100 f and100 c, respectively.

A local device 100 d is being held by the local user LU11. Also, a localdevice 100 e is flying in front of the local device 100 a. These localdevices 100 d and 100 e may be captured by the camera 101 a of the localdevice 100 a. The remote user RU11 recognizes the presence of the localdevices 100 d and 100 e through the video streamed from the local device100 a. At this point, suppose that in one scenario, the remote user RU11selects the local device 100 d as the new streaming source device.Subsequently, the remote user RU11 transfers to the local device 100 d(in actuality, the remote user RU11 still remains at the remotelocation, but the display of the user image of the remote user RU11moves to the display of the local device 100 d). In this state, byhaving the local user LU11 carry around the local device 100 d, theremote user RU11 may move around the local environment 10 virtually.Additionally, suppose that in another scenario, the remote user RU11selects the local device 100 e as the new streaming source device.Subsequently, the remote user RU11 transfers to the local device 100 e(in actuality, the remote user RU11 still remains at the remotelocation, but the display of the user image of the remote user RU11moves to the display of the local device 100 e). In this state, theremote user RU11 remotely operates the local device 100 e, and thus theremote user RU11 may move around the local environment 10 virtually. Forexample, after virtually moving in front of the local device 100 b, theremote user RU11 may participate in conversations with the local usersLU13 and LU14 as well as the remote users RU12 and RU13.

<2. Example Configuration Of Devices>

Next, configurations of the various devices discussed above thatconstitute the information processing system 1 will be described indetail.

[2-1. Local Device]

The local devices 100 a and 100 b are stationary devices installedsemi-permanently (that is, the devices ordinarily do not move in themiddle of remote communication). FIG. 3A is a block diagram illustratingan example of a configuration of the local device 100 configured as astationary device. Referring to FIG. 3A, the local device 100 isequipped with a camera 101, a microphone 103, a display 105, a speaker107, an input unit 109, a network (NW) communication unit 111, ashort-range communication unit 113, a sensor module 115, a storage unit117, and a control unit 120. The control unit 120 may function as astreaming control unit 130, a recognition unit 140, a display controlunit 150, a setting unit 160, and an additional function unit 190. Astationary device is not limited to the examples illustrated in FIGS. 1and 2, and may also be another type of device, such as a televisiondevice, digital signage, or a navigation device, for example.

The local devices 100 c and 100 d are portable devices carried around bylocal users. FIG. 3B is a block diagram illustrating an example of aconfiguration of the local device 100 configured as a portable device.Referring to FIG. 3B, the local device 100 is equipped with a battery195 in addition to the block group illustrated in FIG. 3A. A portabledevice is not limited to the examples illustrated in FIGS. 1 and 2, andmay also be another type of device, such as a notebook PC, a personaldigital assistant (PDA), or a game console, for example. FIG. 4A is anexplanatory diagram illustrating an additional example of a portabledevice. Referring to FIG. 4A, a local device 100 g is a wearable devicein the shape of a T-shirt equipped with a camera 101 g and a display 105g. The local device 100 g is worn by a local user, and may move aroundthe local environment together with that local user.

The local devices 100 e and 100 f are mobile devices that move underoperation by a remote user or autonomously. FIG. 3C is a block diagramillustrating an example of a configuration of the local device 100configured as a mobile device. Referring to FIG. 3C, the local device100 is equipped with a movement mechanism 119 in addition to the blockgroup illustrated in FIG. 3B. The control unit 120 may functionadditionally as a movement control unit 170. A mobile device is notlimited to the examples illustrated in FIGS. 1 and 2, and may also beanother type of device. FIGS. 4B, 4C, and 4D are explanatory diagramseach illustrating an additional example of a mobile device. Referring toFIG. 4B, a local device 100 h is an airship equipped with a camera 101 hand a display 105 h. Referring to FIG. 4C, a local device 100 i is ahumanoid robot equipped with a camera 101 i and a display 105 i.Referring to FIG. 4D, a local device 100 j is a pet robot equipped witha camera 101 j and a display 105 j.

(1) Camera

The camera 101 includes an image sensor such as a charge-coupled device(CCD) or complementary metal-oxide-semiconductor (CMOS) sensor, andcaptures images. A series of images captured by the camera 101constitutes a video. The camera 101 outputs a video signal expressingthe captured video (typically a video depicting the real space of thelocal environment 10) to the control unit 120.

(2) Microphone

The microphone 103 picks up audio emitted near the local device 100, andgenerates an audio signal. The microphone 103 may be a nondirectionalmicrophone, or have directionality. The microphone 103 outputs thegenerated audio signal to the control unit 120.

(3) Display

The display 105 includes a screen realized by a liquid crystal display(LCD), organic light-emitting diodes (OLEDs), a cathode ray tube (CRT),or the like and a display circuit, and displays images under control bythe display control unit 150 discussed later. The display 105 may alsobe a device such as a 3D display, a holographic display, or a volumetricdisplay, instead of a 2D display that displays two-dimensional images.

(4) Speaker

The speaker 107 includes a diaphragm and circuit elements such as anamp, and outputs audio based on an audio signal input from the controlunit 120.

(5) Input Unit

The input unit 109 is an input interface used in order for the localuser to operate the local device 100 or input information into the localdevice 100. The input unit 109 may also include a touch sensorintegrated with the display 105, for example. Instead of (or in additionto) the above, the input unit 109 may also include other types of inputinterfaces, such as buttons, switches, a keyboard, a pointing device, ora keypad. Additionally, the input unit 109 may also include a voicerecognition module that detects user input based on voice commands, or agesture recognition module that detects user input based on gesturecommands.

(6) NW Communication Unit

The NW communication unit 111 is a communication interface that executescommunication on the network 20. The NW communication unit 111 may be awired communication interface that supports a wired communicationprotocol such as Ethernet. Alternatively, the NW communication unit 111may be a wireless communication interface that supports a wirelesscommunication protocol such as Long Term Evolution (LTE), Wideband CodeDivision Multiple Access (WCDMA), WiMAX, or wireless local area network(WLAN).

(7) Short-range Communication Unit

The short-range communication unit 113 is a short-range communicationinterface that supports a short-range communication protocol such asnear field communication (NFC), Bluetooth (registered trademark),TransferJet (registered trademark), or wireless USB. The short-rangecommunication unit 113 executes communication with nearby devicespositioned near the local device 100 in the local environment 10.

(8) Sensor Module

The sensor module 115 is a module that may include a sensor group, suchas a positioning sensor, an acceleration sensor, a gyro sensor, and ageomagnetic sensor. The positioning sensor measures the current positionof the local device 100 based on Global Positioning System (GPS) signalsfrom GPS satellites, or wireless signals from wireless access points,for example. The acceleration sensor measures 3-axis accelerationimparted to the local device 100. The gyro sensor measures the rotationof the local device 100. The geomagnetic sensor measures thegeographical orientation of the local device.

(9) Storage Unit

The storage unit 117 is realized with a storage medium such assemiconductor memory or a hard disk, and stores programs and data usedin processing by the local device 100. The data stored by the storageunit 117 may include local device data about the local device 100, forexample. Note that some of the programs and data described in thisspecification may also be acquired from an external data source (such asa data server, network storage, or externally attached memory, forexample), rather than being stored in the storage unit 117.

(10) Movement Mechanism

The movement mechanism 119 is a means of movement that moves the mobiledevice, namely the local device 100, in real space, under control by themovement control unit 170 discussed later. For example, the movementmechanism 119 may include a motor and a propeller (in the case of aflying type such as a quadcopter or an airship), wheels or caterpillartracks (in the case of a surface movement type such as a talking head),or walking actuators (in the case of a walking type such as a humanoidor pet robot).

(11) Control Unit

The controller 120 may be a processor such as a central processing unit(CPU) or a digital signal processor (DSP). The control unit 120 causesvarious functions of the local device 100 to operate by executing aprogram stored in the storage unit 117 or another storage medium.

(12) Streaming Control Unit

When the current device is selected from among multiple local devicesthat capture a real space as the streaming source device by a remoteuser, the streaming control unit 130 causes a video of the localenvironment 10 captured by the camera 101 to be streamed from the NWcommunication unit 111 to the remote terminal 200. The streaming datastreamed to the remote terminal 200 may also include audio of the localenvironment 10 picked up by the microphone 103, in addition to the videoof the local environment 10. The streaming control unit 130 may deliverthe streaming data to the remote terminal 200 directly, or deliver thestreaming data to the remote terminal 200 via the management server 300.

The streaming control unit 130 starts video streaming to the remoteterminal 200 in response to a streaming request received from themanagement server 300 via the NW communication unit 111, for example. Astreaming request may be received when a remote user logs in to theinformation processing system 1 and selects the local device 100 as theinitial streaming source device. Additionally, a streaming request mayalso be received when the streaming source device is changed from anearby device in the real space to the current device. Video streamingto the remote terminal 200 may be continued until a withdrawalnotification, which means that the remote user is withdrawing from thecurrent device, is received via the NW communication unit 111. Awithdrawal notification may be received when the remote user logs out ofthe information processing system 1, or when the streaming source deviceis changed to a nearby device in real space, for example.

(13) Recognition Unit

The recognition unit 140 recognizes one or more nearby devices in thelocal environment 10 capturing video of the real space that may bestreamed to the remote terminal 200.

In a first technique, the recognition unit 140 may recognize a nearbydevice by detecting a marker displayed by a nearby device in the videocaptured by the camera 101. FIG. 5 illustrates a situation in which twolocal devices 100 d and 100 e appear in an input image IM11, which is aframe constituting a video depicting the local environment 10. The localdevice 100 d is displaying a marker 142 d. The local device 100 e isdisplaying a marker 142 e. A marker detected by the recognition unit 140may be a known figure, symbol, or character string, or may be atwo-dimensional barcode. Information-concealing technology (such assteganography) may also be used to display a marker on the display 105in a format that not perceived by the local user. In addition, a deviceimage of each local device 100 learned in advance may also be used asthe marker. The recognition unit 140 may identify a corresponding devicebased on an image of a detected marker, and recognize the identifieddevice as a nearby device.

In a second technique, the recognition unit 140 may recognize a nearbydevice based on device information received from the nearby device viathe short-range communication unit 113. For example, the recognitionunit 140 of each local device 100 causes device information enablingnearby devices to recognize that the local device 100 is able to streama video of the real space to be broadcast periodically from theshort-range communication unit 113. The device information broadcast atthis point may include a device ID of the local device 100 and otherattribute information, for example. Subsequently, when deviceinformation broadcast from a nearby device in the same way is receivedby the short-range communication unit 113, the recognition unit 140 mayrecognize the device that transmitted the received device information asa nearby device.

The recognition unit 140, after recognizing a nearby device according toeither of the above techniques (or another technique), notifies theremote terminal 200 of the presence of the recognized nearby device.Subsequently, if the remote user decides to change the streaming sourcedevice, the withdrawal notification discussed above may be received bythe NW communication unit 111. The notification of a nearby device tothe remote terminal 200 may also be performed indirectly via themanagement server 300. For example, the management server 300 may notifythe remote terminal 200 of the presence of a nearby device capable ofstreaming a video after first validating the compatibility of thestreaming protocol between the recognized nearby device and the remoteterminal 200.

Note that the recognition of a nearby device may also be conducted bythe remote terminal 200, rather than the recognition unit 140 of thelocal device 100. For example, the remote terminal 200 may recognize anearby device to the local device 100 by detecting a marker appearing ina video streamed from the local device 100.

(14) Display Control Unit

While a video of the real space is being streamed to the remote terminal200, the display control unit 150 causes the display 105 to display auser image of the remote user who is viewing that video. The image datafor the user image of the remote user may be included in the streamingrequest received by the NW communication unit 111, or acquired from adatabase using the user ID of the remote user. In addition, the displaycontrol unit 150 may also cause the display 105 to display userattributes, such as a nickname of the remote user, together with theuser image.

Additionally, when the streaming source device for video streaming tothe remote terminal 200 is changed from the current device to a nearbydevice, the display control unit 150 may cause the display 105 todisplay an animation of the remote user transferring to the nearbydevice. FIG. 6 is an explanatory diagram illustrating an example ofdisplay image transitions in the local device 100 when the remote usertransfers to a nearby device (in other words, the selected streamingsource device is changed to a nearby device). At time t=T1, the localdevice 100 a is the streaming source device, and as illustrated on theleft side of FIG. 6, a user image 152 a of the remote user is beingdisplayed on the display 105 a of the local device 100 a. After that,suppose that the remote user changes the streaming source device to anearby device 100 e of the local device 100 a. At time t=T2, there isdisplayed on the display 105 a an image 154 a constituting an animationas though the remote user is flying out from the local device 100 a.After this animation ends, at time t=T3, the local device 100 a is nolonger the streaming source device, and a marker 142 a is beingdisplayed on the display 105 a.

Additionally, when the streaming source device for video streaming tothe remote terminal 200 is changed from a nearby device to the currentdevice, the display control unit 150 may cause the display 105 todisplay an animation of the remote user transferring from the nearbydevice to the current device. FIG. 7 is an explanatory diagramillustrating an example of display image transitions in the local device100 when the remote user transfers from a nearby device (in other words,the selected streaming source device is changed to the current device).At time t=T1, the local device 100 e is a nearby device from theperspective of the streaming source device, which is another device. Atthis time, as illustrated on the left side of FIG. 7, a marker 142 e isbeing displayed on the display 105 e of the local device 100 e. Afterthat, suppose that the remote user changes the streaming source deviceto the local device 100 e. At time t=T2, there is displayed on thedisplay 105 e an image 154 e constituting an animation as though theremote user is going into the local device 100 e. After this animationends, at time t=T3, the local device 100 e is the streaming sourcedevice, and a user image 152 e of the remote user is being displayed onthe display 105 e.

Other local devices 100 not involved in the change may also be notifiedof the change of streaming source device. If at least one of thestreaming source device from before the change or after the changeappears in an input image, the display control unit 150 of the localdevice 100 receiving such a notification may also cause the display 105to display an animation of the remote user transferring between the twodevices. FIG. 8 is an explanatory diagram illustrating an example of adisplay image on a local device 100 that takes a picture of two deviceswhen a remote user transfers between the two devices. In the image IM12illustrated in FIG. 8, the display 105 a of the local device 100 a andthe display 105 e of the local device 100 e appear. The local device 100a is the streaming source device before the change. The local device 100e is the streaming source device after the change. In the image IM12displayed at time t=T2, an image 154 d constituting an animation asthough the remote user is flying out from the local device 100 a andgoing into the local device 100 e is displayed.

By viewing animations like those illustrated as examples in FIGS. 6 to8, the local user is able to recognize easily and intuitively whichdevice among the local devices 100 the remote user is residing in (whichdevice from which the remote user is viewing a video), and also fromwhich device to which device the remote user transferred.

As FIGS. 6 and 7 demonstrate, the display control unit 150 causes thedisplay 105 to display a user image of the remote user residing in thecurrent device. If multiple remote users are residing in the currentdevice, or in other words, if the video is being streamed to multipleremote terminals 200 respectively used by multiple remote users, thedisplay control unit 150 may also cause the display 105 to display theuser images of the multiple remote users, as illustrated in FIG. 9. Inthe example of FIG. 9, the display image IM13 is displaying the userimages of remote users RU11, RU12, and RU14.

If a remote user is not residing in the current device, or if there isremaining capacity to accept a new remote user, the display control unit150 may cause the display 105 to display the marker discussed above. Themarker is used to enable nearby devices to recognize that the currentdevice is capable of streaming a video of the real space. Note that inthe case of using a device image of the exterior appearance of the localdevice 100 as the marker, the display control unit 150 may also notperform the marker display.

(15) Setting Unit

The setting unit 160 configures settings of the local device 100according to user input from a local user present in the real space. Forexample, the setting unit 160 measures the current position of the localdevice 100 based on sensor data from the sensor module 115, and reportsthe positioning result to the management server 300. Additionally, thesetting unit 160 configures whether or not to allow streaming of a videoof the local environment 10 to the remote terminal 200, according touser input detected via the input unit 109. If streaming is not allowed,the streaming control unit 130 does not stream the video captured by thecamera 101. The allowance or disallowance of streaming may be configuredin arbitrary units, such as the address of the remote terminal 200, theuser ID of the remote user, or a group to which the remote user belongs.For example, one user may be allowed to access all local devices 100,whereas another user may be allowed to access only local devices 100 ofa specific type (for example, stationary devices). According to suchsettings, security risks caused by various remote users freely movingaround the local environment 10 may be avoided.

(16) Movement Control Unit

The movement control unit 170 is a function block provided in the localdevice 100 in the case in which the local device 100 is a mobile deviceincluding the movement mechanism 119. The movement control unit 170controls the movement mechanism 119 according to a movement controlsignal received from the remote terminal 200 during the period in whichthe remote user is residing in the local device 100. The movementcontrol signal may be a signal instructing the local device 100 to goforward, go backward, accelerate, decelerate, change direction, or (ifpossible) ascend or descend, for example. Additionally, during theperiod in which the remote user is not residing in the local device 100,the movement control unit 170 may also control the movement mechanism119 autonomously according to some kind of algorithm, so that the localdevice moves around the local environment 10.

If multiple remote users are residing in the local device 100 asdescribed using FIG. 9 (that is, if the video captured by the camera 101is being streamed to multiple remote terminals 200), there is apossibility that the movement control signals received from these remoteterminals 200 may conflict with each other. For example, if twodifferent remote users respectively instruct the same local device 100to go forward and go backward at the same time, these instructions willconflict with each other. To resolve such conflicts, the movementcontrol unit 170 may decide the motion of the local device 100 based onthe movement control signal received from at least one of the multipleremote terminals 200, in accordance with a designated condition. Herein,the designated condition may be obeying control by a master user fromamong the multiple remote users, basing the decision on a majority voteby the multiple remote users, or pausing control when conflictingcontrol instructions are given. In the example of FIG. 9, the user imageof the master user, namely the remote user RU11, is displayed largerthan the user images of the other remote users.

(17) Additional Function Unit

The additional function unit 190 provides additional functions to users,depending on the capabilities of the local device 100. The additionalfunctions provided by the additional function unit 190 may include, forexample, a graphic drawing function (a function of displaying, on thedisplay 105, graphics drawn on the remote terminal 200 by the remoteuser). Also, the additional functions may include a content playbackfunction (a function of playing back content on the local device 100under remote control from the remote terminal 200). Note that thesefunctions are merely examples, and other functions may also be providedby the additional function unit 190. A list of the additional functionsprovidable by each local device 100 may be stated in the local devicedata, for example.

(18) Battery

In the case in which the local device 100 is a portable device or amobile device, the local device 100 may be equipped with the battery195. The battery 195 supplies operating power to each block in thedrawings via power supply lines partially illustrated with dashed linesin FIGS. 3B and 3C. If the remaining battery level falls below athreshold value, the local device 100 may also switch to a power-savingmode. In the power-saving mode, the streaming control unit 130 may alsopause video streaming. Also, in the power-saving mode, the movement of amobile device may be prohibited.

[2-2. Remote Terminal]

FIG. 10 is a block diagram illustrating an example of a configuration ofthe remote terminal 200. Referring to FIG. 10, the remote terminal 200is equipped with a camera 201, a microphone 203, a display 205, aspeaker 207, an input unit 209, a NW communication unit 211, a storageunit 217, and a control unit 220. The control unit 220 may function as astreaming control unit 230, a recognition unit 240, and a device controlunit 250.

(1) Camera

The camera 201 includes an image sensor such as a CCD or CMOS sensor,and captures images. Images captured by the camera 201 may include auser image of the remote user RU, for example. The camera 201 outputs animage signal expressing a captured image to the control unit 220.

(2) Microphone

The microphone 203 picks up audio emitted by the remote user, andgenerates an audio signal. The microphone 203 outputs the generatedaudio signal to the control unit 220.

(3) Display

The display 205 includes a screen realized by a LCD, OLED, CRT, or thelike and a display circuit, and displays images. Images displayed by thedisplay 205 may include graphical user interface (GUI) images discussedlater, for example.

(4) Speaker

The speaker 207 includes a diaphragm and circuit elements such as anamp, and outputs audio based on an audio signal input from the controlunit 220.

(5) Input Unit

The input unit 209 is an input interface used in order for the remoteuser to operate the remote terminal 200 or input information into theremote terminal 200. The input unit 209 may also include a touch sensorintegrated with the display 205, for example. Instead of (or in additionto) the above, the input unit 209 may also include other types of inputinterfaces, such as buttons, switches, a keyboard, a pointing device, ora keypad. Additionally, the input unit 209 may also include a voicerecognition module that detects user input based on voice commands, or agesture recognition module that detects user input based on gesturecommands.

(6) NW Communication Unit

The NW communication unit 211 is a communication interface that executescommunication on the network 20. The NW communication unit 211 may be awired communication interface that supports a wired communicationprotocol such as Ethernet. Alternatively, the NW communication unit 211may be a wireless communication interface that supports a wirelesscommunication protocol such as LTE, WCDMA, WiMAX, or WLAN. The NWcommunication unit 211 may communicate with the multiple local devices100 discussed earlier that respectively capture videos of a real space,either directly or via the management server 300.

(7) Storage Unit

The storage unit 217 is realized with a storage medium such assemiconductor memory or a hard disk, and stores programs and data usedin processing by the remote terminal 200. The data stored by the storageunit 217 may include remote user data about the remote user who uses theremote terminal 200, for example. Note that some of the programs anddata described in this specification may also be acquired from anexternal data source, rather than being stored in the storage unit 217.

(8) Control Unit

The control unit 220 may be a processor such as a CPU or DSP. Thecontrol unit 220 causes various functions of the remote terminal 200 tooperate by executing a program stored in the storage unit 217 or anotherstorage medium.

(9) Streaming Control Unit

The streaming control unit 230 causes the NW communication unit 211 toreceive streaming data delivered from one streaming source deviceselected by the remote user from among multiple local devices 100present in the local environment 10. Subsequently, the streaming controlunit 230 uses the display 205 to play back video included in thestreaming data. The video played back at this point is a video depictingthe real space of the local environment 10. Additionally, the streamingcontrol unit 230 may also use the speaker 207 to play back audioincluded in the streaming data. Additionally, the streaming control unit230 may also execute streaming from the remote terminal 200 to thestreaming source device. The streaming data streamed from the remoteterminal 200 may include images captured by the camera 201 and audiopicked up by the microphone 203. Streaming between the remote terminal200 and the streaming source device may be conducted directly, or may berelayed by the management server 300.

(10) Recognition Unit

The recognition unit 240 recognizes one or more nearby devices of thestreaming source device appearing in video played back by the streamingcontrol unit 230. Herein, a nearby device refers to a local device 100capable of streaming to the remote terminal 200 a video of the realspace from a viewpoint different from the currently playing video. Forexample, the recognition unit 240 may recognize a nearby device bydetecting the above-discussed marker displayed by a nearby device in avideo that is played back. The recognition unit 240 may identify acorresponding device based on an image of a detected marker, andrecognize the identified device as a nearby device. Subsequently, therecognition unit 240 notifies the device control unit 250 discussedlater of the presence of the recognized nearby device. Note that therecognition of a nearby device may also be conducted entirely by therecognition unit 140 of the local device 100. In this case, therecognition unit 240 may be omitted from the configuration of the remoteterminal 200.

(11) Device Control Unit

The device control unit 250 presents the remote user with informationabout local devices 100 in the local environment 10 that are capturingvideo streamable to the remote terminal 200, enabling the remote user toselect a streaming source device. The streaming source device when theremote user logs in to the information processing system 1 may beselected by the remote user at every login, or be a default device setin advance. After a streaming source device is selected, the devicecontrol unit 250 causes the streaming control unit 230 to play backvideo received from the selected streaming source device. Additionally,while video from the streaming source device is being played back by thestreaming control unit 230, the device control unit 250 causes thedisplay of the streaming source device to display a user image of theremote user.

Once a streaming source device has been selected, if one or more nearbydevices are recognized by the recognition unit 240 or the recognitionunit 140 of the streaming source device, the device control unit 250asks the remote user about whether to change the streaming source deviceto one of the nearby devices. Asking the remote user may be performedvia a GUI displayed by the display 205, or via audio output by thespeaker 207. When asking the user whether to change the streaming sourcedevice, the device control unit 250 may also present the remote userwith information related to the functionality of recognized nearbydevices.

FIG. 11 is an explanatory diagram illustrating an example of a GUI thatmay be displayed on the remote terminal 200. Referring to FIG. 11, a GUIimage IM2 that may be generated by the device control unit 250 isillustrated. The GUI image IM2 includes a streaming window W21, aprofile window W22, a device information window W23, a location windowW24, and a history window W25. The streaming window W21 displays videoplayed back by the streaming control unit 230. In the example of FIG.11, the local device 100 e appears in the video received from thestreaming source device. The profile window W22 displays a profile ofthe remote user. In the example of FIG. 11, a user image of the remoteuser (the same image as the one displayed on the display of thestreaming source device), a nickname, and the name of a group to whichthe remote user belongs are being displayed in the profile window W22.The device information window W23 displays information related to thecurrently selected streaming source device. In the example of FIG. 11,the name, type, and functionality of the local device 100 a are beingdisplayed by the device information window W23. Such information may beprovided to the remote terminal 200 from the management server 300 (orfrom the streaming source device), for example. The location window W24displays a map indicating the current position in the local environment10 of the selected streaming source device. The history window W25displays a history of streaming source device selections, or in otherwords, a list of previously selected streaming source devices.

In the streaming window W21 of FIG. 11, display objects D21 and D22 areoverlaid. The display object D21 points out the marker of the localdevice 100 e detected in the video being streamed. The local device 100e is a nearby device of the streaming source device. The display objectD22 is an information area displaying device information about the localdevice 100 e recognized as a nearby device. In the example of FIG. 11,the name, type, and functionality of the local device 100 e are beingdisplayed by the display object D22. In addition, the display object D22may also indicate information about a user residing in the local device100 e. However, in the example of FIG. 11, there are no users residingin the local device 100 e (“None”). Furthermore, the display object D22includes a button D23. The remote user, by pressing the button D23 viathe input unit 209 (such as clicking or tapping), is able to select thelocal device 100 e as the new streaming source device.

After a new streaming source device is selected by the remote user viasuch GUI image, the device control unit 250 transmits a device changerequest to the management server 300, for example. The management server300, in response to receiving the device change request from the remoteterminal 200, may transmit a streaming request to the local device 100selected as the new streaming source device, and may transmit awithdrawal notification to the current streaming source device.Alternatively, the device control unit 250 may also transmit thestreaming request and the withdrawal notification directly to therespective destination local devices 100. Note that if a devicedisplayed in the history window W25 is specified by the remote user, thedevice control unit 250 may change the streaming source device to thespecified device.

If the streaming source device is a mobile device including a movementmechanism, while video from the streaming source device is being playedback, the device control unit 250 may also generate a movement controlsignal for controlling the movement of the streaming source deviceaccording to user input from the remote user. As discussed earlier, themovement control signal may be a signal instructing the streaming sourcedevice to go forward, go backward, accelerate, decelerate, changedirection, or (if possible) ascend or descend, for example. The movementcontrol signal generated by the device control unit 250 may betransmitted to the streaming source device over the network 20 via theNW communication unit 211. For example, the remote user, aftertransferring to a mobile device, is able to move the mobile device to alocation where communication between local users is taking place (themovement in this case is not virtual), and participate in thecommunication. Additionally, after moving the mobile device near anotherlocal device 100, the remote user is also able to transfer further tothe other local device 100.

[2-3. Management Server]

FIG. 12 is a block diagram illustrating an example of a configuration ofthe management server 300. Referring to FIG. 12, the management server300 is equipped with an input unit 309, a NW communication unit 311, astorage unit 317, and a control unit 320. The control unit 320 mayfunction as a data management unit 330 and a connection management unit340.

(1) Input Unit

The input unit 309 is an input interface used in order for an operatorto operate the management server 300 or input information into themanagement server 300. The input unit 309 may include arbitrary types ofinput interfaces, such as buttons, switches, a keyboard, a pointingdevice, or a keypad.

(2) NW Communication Unit

The NW communication unit 311 is a communication interface that executescommunication on the network 20. The NW communication unit 311 may be awired communication interface or a wireless communication interface. TheNW communication unit 311 may communicate with the multiple localdevices 100 and the remote terminal 200 discussed earlier.

(3) Storage Unit

The storage unit 317 is realized with a storage medium such assemiconductor memory or a hard disk, and stores programs and data usedin processing by the management server 300. The data stored by thestorage unit 317 may include local device data 350 and remote user data360 discussed later, for example.

(4) Control Unit

The control unit 320 may be a processor such as a CPU or DSP. Thecontrol unit 320 causes various functions of the management server 300to operate by executing a program stored in the storage unit 317 oranother storage medium.

(5) Data Management Unit

The data management unit 330 has a role of acting as a manager thatmanages data stored by the storage unit 317. When a new local device 100is deployed in the local environment 10, the data management unit 330adds a record corresponding to the device in the local device data 350.Additionally, when a new remote user is registered, the data managementunit 330 adds a record corresponding to the user in the remote user data360. Additionally, the data management unit 330 keeps the content ofthis data in the latest state through the exchange of messages with thelocal devices 100 and the remote terminal 200.

FIG. 13 is an explanatory diagram illustrating an example of a structureof the local device data 350 managed on the management server 300.Referring to FIG. 13, the local device data 350 includes data fields fordevice ID 351, name 352, type 353, functionality 354, address 355,position 356, access rights 357, and status 358. The device ID 351 is anidentifier that uniquely identifies each device. The name 352 expressesa name given to each device. The type 353 expresses the device type ofeach device. As a non-limiting example, the device type may be any ofthe three types of “Stationary”, “Portable”, and “Mobile”. Thefunctionality 354 is a data field stating a list of additional functionsprovidable by each device. The address 355 expresses a destinationaddress to be inserted into messages transmitted to each device (forexample, an IP address or a MAC address). The position 356 expressescoordinates indicating the current position of each device. The datamanagement unit 330 may update the corresponding position coordinatesbased on a positioning result report received periodically from eachdevice, for example. The access rights 357 state access rightsinformation for managing remote user access to each device. In theexample of FIG. 13, access to the devices with the device IDs “DEV_A”and “DEV_C” is granted to all users (“All Users”). On the other hand,access to the device with the device ID “DEV_F” is granted only to usersbelonging to a specific group (“Group A”). Such access rightsinformation may be preregistered by the operator of the managementserver 300, or registered dynamically through the setting unit 160 ofeach local device 100. The status 358 expresses the operating status ofeach device.

The data management unit 330 provides a subset of the above local devicedata 350 to the remote terminal 200. In the device information windowW23 of the GUI image IM2 described using FIG. 11, a subset about thestreaming source device is displayed as device information. In thedisplay object D22, a subset about the nearby device is displayed asdevice information to assist in the selection of a new streaming sourcedevice by the remote user.

FIG. 14 is an explanatory diagram illustrating an example of a structureof the remote user data 360 managed on the management server 300.Referring to FIG. 14, the remote user data 360 includes data fields foruser ID 361, nickname 362, resident device 363, terminal ID 364,terminal address 365, and user image 366. The user ID 361 is anidentifier that uniquely identifies each remote user. The nickname 362expresses a nickname given to each remote user. The resident device 363expresses a device ID of the streaming source device currently selectedby each remote user. The data management unit 330 may update theresident device 363 ever time a remote user transfers from one device toanother device. The terminal ID 364 is an identifier that identifies theremote terminal 200 being used by each remote user. The terminal address365 expresses an address of the remote terminal 200 being used by eachremote user (for example, an IP address or a MAC address). The userimage 366 is a data field that stores image data of a user image foreach remote user.

(6) Connection Management Unit

The connection management unit 340 manages connections between localdevices 100 and remote terminals 200. The connection management unit 340receives a login request from the remote terminal 200, and if the remoteuser is successfully authenticated, approves the login request, forexample. Remote user authentication may be simple authentication basedon information such as an ID and a password, or the IP address of theterminal, or may be more advanced authentication based on informationsuch as an encryption key. The connection management unit 340 decides aninitial device to act as the first streaming source device for theremote user whose login is approved, and transmits a streaming requestto the decided initial device. The streaming request may include remoteuser data about the remote user who is to view the streamed video. Theinitial device may be a specific local device 100 in the localenvironment 10, or a local device 100 predefined for each remote user.Alternatively, the remote user may also select the initial device.

If a nearby device of an already-selected streaming source device isselected by the remote user as the new streaming source device, theconnection management unit 340 transmits a streaming request to the newstreaming source device in response to a device change request that maybe received from the remote terminal 200. The connection management unit340 may also verify the compatibility of the protocol between the nearbydevice and the remote terminal 200 before coordinating the newconnection. Additionally, the connection management unit 340 may alsoverify whether or not access to the nearby device from the remoteterminal 200 is allowed, based on the access rights information for thenearby device. The connection management unit 340 may transmit awithdrawal notification to the streaming source device from before thechange.

If the streaming request is approved, the connection management unit 340transmits the local device data of the streaming source device thatapproved the request to the remote terminal 200. After that, thestreaming of video (and audio) from the streaming source device to theremote terminal 200 is started. In addition, a user image of the remoteuser is displayed on the streaming source device. The connectionmanagement unit 340 may also relay the streaming data transmitted fromthe streaming source device to the streaming destination remote terminal200.

<3. Process Flows>

[3-1. Login to System]

FIG. 15 is a sequence diagram illustrating an example of a flow ofinformation processing during login of a remote user into a system.

Referring to FIG. 15, first, the remote terminal 200 transmits a loginrequest to the management server 300 (step S5). The management server300 authenticates the remote user (step S10), and returns anauthentication result to the remote terminal 200 (step S15). At thispoint, suppose that the authentication of the remote user is successful,and the login is approved. After the login is approved, the remoteterminal 200 transmits user data to the management server 300 (stepS20). The user data transmitted at this point may include the latestinformation about the remote user (such as a user image of the remoteuser taken during login, for example).

The management server 300 decides the initial device from among themultiple local devices 100 present in the local environment 10 for theremote user whose login is approved (step S25). Subsequently, themanagement server 300 transmits a streaming request to the decidedinitial device (step S30). The local device 100 decided as the initialdevice replies to the management server 300 with a response indicatingwhether or not streaming is available (step S35). At this point, supposethat the local device 100 determines that streaming is available (basedon factors such as the capacity of the device, for example). If themanagement server 300 receives a response indicating that the streamingrequest has been accepted, the management server 300 transmits deviceinformation about the streaming source device (a subset of the localdevice data 350) to the remote terminal 200 (step S40).

The local device 100, after replying to the management server 300 withthe response to the streaming request, causes its display to display auser image of the remote user (step S50). The remote terminal 200 causesits display to display the device information provided by the managementserver 300 (step S60). The management server 300 updates the data storedin the database (step S65). After that, the streaming of video (andaudio) between the local device 100 and the remote terminal 200 isstarted (step S70).

[3-2. Transfer Between Devices]

FIG. 16 is a sequence diagram illustrating an example of a flow ofinformation processing when a remote user transfers from a local deviceto a nearby device. Suppose that at the beginning of the sequence inFIG. 16, the local device 100 a has been selected as the streamingsource device. The local device 100 e is a nearby device positioned nearthe local device 100 a.

Referring to FIG. 16, first, the streaming of video (and audio) betweenthe local device 100 a and the remote terminal 200 is executed (stepS100). After that, in the local device 100 a or in the remote terminal200, nearby devices of the local device 100 a are recognized (steps S105a, S105 b). The nearby device recognition results are reported to themanagement server 300. For a recognized nearby device, the managementserver 300 verifies the connectivity with the remote terminal 200 (forexample, protocol compatibility, and whether or not access is allowed)(step S110). At this point, suppose that the local device 100 e isrecognized as a nearby device, the connectivity between the local device100 e and the remote terminal 200 is affirmed. The management server 300transmits device information about the verified nearby device, namelythe local device 100 e, to the remote terminal 200 (step S115).

The remote terminal 200 causes its display to display the deviceinformation received from the management server 300, and asks the userwhether to transfer to the local device 100 e (in other words, whetherto change the streaming source device to the local device 100 e) (stepS120). Subsequently, if user input giving instructions to change thestreaming source device is detected (step S125), the remote terminal 200transmits a device change request to the management server 300 (stepS130). The management server 300, in response to receiving the devicechange request from the remote terminal 200, transmits a streamingrequest to the local device 100 e (step S140). The local device 100 ereplies to the management server 300 with a response indicating whetheror not streaming is available (step S145). At this point, suppose thatthe local device 100 e determines that streaming is available. Themanagement server 300 additionally transmits a withdrawal notificationto the local device 100 a (step S150). The local device 100 a replies tothe management server 300 with an acknowledgement (ACK) of thewithdrawal notification (step S155). The management server 300, afterreceiving from the local device 100 e a response indicating that thestreaming request has been accepted, replies to the remote terminal 200with a response indicating that the device change is allowed (stepS160).

After that, the local device 100 e that accepted the streaming requestdisplays an animation of the remote user transferring to the deviceitself, as illustrated by example in FIG. 7 (step S165). Additionally,the local device 100 a that received the withdrawal notificationdisplays an animation of the remote user transferring from the deviceitself to a nearby device, as illustrated by example in FIG. 6. Theanimation of the remote user transferring to a nearby deviceadditionally may be displayed on the remote terminal 200 (step S170).

After the display of the animation discussed above ends, the localdevice 100 e causes its display to display a user image of the remoteuser (step S175). Meanwhile, the local device 100 a removes a user imageof the remote user from its display (step S176). The remote terminal 200causes its display to display the device information about the localdevice 100 e provided by the management server 300 (step S180). Themanagement server 300 updates the data stored in the database (stepS185). After that, the streaming of video (and audio) between the localdevice 100 e and the remote terminal 200 is started (step S190).Meanwhile, the streaming between the local device 100 a and the remoteterminal 200 is stopped.

<4. Conclusion>

The foregoing thus describes an embodiment of technology according tothe present disclosure in detail using FIGS. 1 to 16. According to theembodiment discussed above, multiple local devices that capture video ofa real space in a local environment are deployed in the real space.Additionally, a local device selected by a remote user from among thesemultiple local devices is made to stream video captured by a camera ofthe device to a remote terminal used by the remote user, and while suchstreaming is being conducted, a user image of the remote user isdisplayed on the display of the device. Consequently, the remote user isable to view the state of the local environment from the viewpoint of adesired device from among various local devices present in the localenvironment. Meanwhile, a local user in the local environment is able torecognize, by looking at the user image being displayed on the localdevice, which viewpoint from which the remote user is viewing the stateof the local environment. Additionally, the remote user is able to movethe viewpoint flexibly from one local device to another local device,enabling face-to-face communication between the remote user and localusers.

In addition, according to the embodiment discussed above, several localdevices are realized as mobile devices including a movement mechanism.Furthermore, the movement mechanism of a mobile device is controlledaccording to a movement control signal received from a remote terminal.Consequently, by selecting a mobile device as the streaming sourcedevice, the remote user is able to control the mobile device whileviewing video streamed from the mobile device, and freely move theviewpoint of the video around the local environment. For example, theremote user is able to move the mobile device to a location wheredesired communication peers are present (such as a conference room, forexample), and conduct remote communication with the local users in thatlocation.

In addition, according to the embodiment discussed above, the presenceof a nearby device near the local device streaming video to the remoteterminal is recognized in the remote terminal, or indicated in anotification from the local device to the remote terminal. Consequently,the remote user is able to serially change the streaming source devicefrom the currently selected local device to a nearby device of the localdevice. By repeatedly changing the selection in this way, a unique userexperience is provided, almost as though a ghost of the remote user istransferring freely from one device to another device in the localenvironment. When the streaming source device is changed, a display ofthe local device may display an animation of the remote usertransferring between devices. In so doing, a local user is able toeasily and intuitively recognize the movement of the viewpoint of theremote user. Consequently, it is possible to prevent a psychologicalburden on a local user which may arise due to a difficulty in knowingfrom where the remote user is looking.

In addition, according to the embodiment discussed above, when a nearbydevice is recognized, the remote terminal provides a GUI for asking theremote user whether or not the streaming source device should be changedto the nearby device. Through such a GUI, the remote user is able toselect the streaming source device easily. The GUI provided by theremote terminal may present information related to the functionality ofthe nearby device to the remote user. In this case, the remote user isable to select a device with desired functionality as the streamingsource device. For example, after first transferring to a mobile device,the remote user is able to search for a desired device by moving theviewpoint of the video around the local environment, and additionallytransfer from the mobile device to a discovered desired device.

Further, a series of control processes by the respective devicesdescribed in the present description may be implemented using any one ofsoftware, hardware, and a combination of hardware and software. Forexample, a program configuring software is stored in a storage medium (anon-transitory medium) installed inside or outside each device. Further,for example, each program is read onto a Random Access Memory (RAM) atthe time of execution and executed by a processor such as a CentralProcessing Unit (CPU).

Further, the processes described using sequence diagrams in the presentdescription may not necessarily be executed in the order indicated bythe sequence diagrams. Some process steps may be executed in parallel.Further, additional process steps may be employed, and some processsteps may be omitted.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art based on the description of this specification.

Additionally, the present technology may also be configured as below.

(1)

An information processing device, including:

a camera that captures a real space;

a communication unit that communicates with a terminal device used by aremote user; a streaming control unit that streams a first videocaptured by the camera from the communication unit to the terminaldevice if the information processing device itself is selected by theremote user from among a plurality of devices that capture the realspace; and

a display that displays a user image of the remote user while the firstvideo is being streamed to the terminal device.

(2)

The information processing device according to (1), further including:

a movement mechanism that causes the information processing device tomove around the real space; and

a movement control unit that controls the movement mechanism accordingto a movement control signal received from the terminal device.

(3)

The information processing device according to (1) or (2), furtherincluding:

a recognition unit that recognizes a first nearby device in the realspace that captures a second video streamable to the terminal device,wherein

if the first nearby device is recognized, the recognition unit notifiesthe terminal device of a presence of the first nearby device.

(4)

The information processing device according to (3), wherein

the recognition unit recognizes the first nearby device by detecting, inthe first video, a marker displayed by the first nearby device.

(5)

The information processing device according to (3), wherein

the information processing device includes a short-range communicationunit, and

the recognition unit recognizes the first nearby device based oninformation received from the first nearby device through theshort-range communication unit.

(6)

The information processing device according to any one of (3) to (5),wherein

when a streaming source device of video streaming to the terminal deviceis changed from the information processing device to the first nearbydevice, the display displays an animation of the remote usertransferring to the first nearby device.

(7)

The information processing device according to any one of (1) to (6),wherein

if a streaming source device of video streaming to the terminal deviceis changed from a second nearby device in the real space to theinformation processing device, the streaming control unit causes thefirst video to be streamed to the terminal device.

(8)

The information processing device according to (7), further including:

a display control unit that causes the display to display a markerindicating that the information processing device is able to stream thefirst video.

(9)

The information processing device according to (7), further including:

a short-range communication unit that transmits information indicatingthat the information processing device is able to stream the firstvideo.

(10)

The information processing device according to (7), wherein

when a streaming source device of video streaming to the terminal deviceis changed from the second nearby device to the information processingdevice, the display displays an animation of the remote usertransferring from the second nearby device to the information processingdevice.

(11)

The information processing device according to any one of (1) to (10),further including:

a setting unit that sets whether or not to allow streaming of the firstvideo to the terminal device, according to user input from a local userpresent in the real space.

(12)

The information processing device according to (2), wherein

if the first video is being streamed to a plurality of terminal devicesrespectively used by a plurality of remote users, the movement controlunit decides a motion of the information processing device based on amovement control signal received from at least one of the plurality ofterminal devices, according to a certain condition.

(13)

An information processing method executed by an information processingdevice provided with a camera that captures a real space and acommunication unit that communicates with a terminal device used by aremote user, the information processing method including:

streaming a first video captured by the camera from the communicationunit to the terminal device if the information processing device itselfis selected by the remote user from among a plurality of devices thatcapture the real space; and

displaying a user image of the remote user on a display while the firstvideo is being streamed to the terminal device.

(14)

A terminal device, including:

a communication unit that communicates with a plurality of devices thatrespectively capture a real space;

a streaming control unit that plays back a first video depicting thereal space streamed from a device selected by a remote user using theterminal device from among the plurality of devices; and

a device control unit that causes a display of the selected device todisplay a user image of the remote user while the first video is beingplayed back by the streaming control unit.

(15)

The terminal device according to (14), wherein

if a first nearby device in the real space that captures a second videostreamable to the terminal device is recognized, the device control unitasks a user whether or not to change a streaming source device of videostreaming to the terminal device to the first nearby device.

(16)

The terminal device according to (15), wherein

when asking a user whether or not to change the streaming source deviceto the first nearby device, the device control unit presents informationrelated to a functionality of the first nearby device to the remoteuser.

(17)

The terminal device according to (15) or (16), further including:

a recognition unit that recognizes the first nearby device appearing inthe first video.

(18)

The terminal device according to (17), wherein

the recognition unit recognizes the first nearby device by detecting, inthe first video, a marker displayed by the first nearby device.

(19)

The terminal device according to any one of (14) to (18), wherein

the plurality of devices includes at least one mobile device including amovement mechanism, and

if the mobile device is selected as the streaming source device, thedevice control unit generates a movement control signal transmitted tothe mobile device to control a movement of the mobile device, accordingto user input from the remote user.

(20)

An information processing method executed by a terminal device providedwith a communication unit that communicates with a plurality of devicesthat respectively capture a real space, the information processingmethod including:

playing back a first video depicting the real space streamed from adevice selected by a remote user using the terminal device from amongthe plurality of devices; and

causing a display of the selected device to display a user image of theremote user while the first video is being played back.

REFERENCE SIGNS LIST

-   1 information processing system-   10 local environment-   20 network-   100 information processing device (local device)-   101 camera-   105 display-   111 network (NW) communication unit-   113 short-range communication unit-   119 movement mechanism-   130 streaming control unit-   140 recognition unit-   150 display control unit-   160 setting unit-   170 movement control unit-   200 terminal device (remote terminal)-   211 network (NW) communication unit-   230 streaming control unit-   240 recognition unit-   250 device control unit-   300 management server

The invention claimed is:
 1. An information processing device,comprising: a camera configured to capture a real space; a communicationunit configured to communicate with a terminal device associated with aremote user; a streaming control unit configured to stream a firstvideo, captured by the camera, from the communication unit to theterminal device based on the information processing device that isselected by the remote user, wherein the information processing deviceis selected from a plurality of devices that capture the real space; adisplay screen configured to display an image of the remote user basedon the first video being streamed to the terminal device; and arecognition unit configured to: determine a first nearby device of theplurality of devices in the real space, wherein the first nearby devicecaptures a second video streamable to the terminal device; and notify apresence of the first nearby device to the terminal device based on adetermination of the first nearby device.
 2. The information processingdevice according to claim 1, further comprising: a movement mechanismconfigured to control a movement of the information processing device inthe real space; and a movement control unit configured to control themovement mechanism based on a movement control signal received from theterminal device.
 3. The information processing device according to claim1, wherein the recognition unit is further configured to determine thefirst nearby device based on a detection of a marker, wherein the markeris displayed by the first nearby device in the first video.
 4. Theinformation processing device according to claim 1, further comprises ashort-range communication unit configured to receive device informationfrom the first nearby device, and wherein the recognition unit isfurther configured to determine the first nearby device based on thereceived device information.
 5. The information processing deviceaccording to claim 1, wherein the display screen is further configuredto display an animation that depicts a transfer of the remote user tothe first nearby device, based on a change of a streaming source devicefrom the information processing device to the first nearby device, andwherein the streaming source device streams the first video or thesecond video to the terminal device.
 6. The information processingdevice according to claim 1, wherein the streaming control unit isfurther configured to stream the first video to the terminal devicebased on a change of a streaming source device from a second nearbydevice of the plurality of devices to the information processing device,wherein the streaming source device streams the first video or thesecond video to the terminal device.
 7. The information processingdevice according to claim 6, further comprising: a display control unitconfigured to control the display screen to display a marker, whereinthe marker indicates an ability of the information processing device tostream the first video.
 8. The information processing device accordingto claim 6, further comprising a short-range communication unitconfigured to transmit information indicating an ability of theinformation processing device to stream the first video.
 9. Theinformation processing device according to claim 6, wherein the displayscreen is further configured to display an animation that depicts atransfer of the remote user from the second nearby device to theinformation processing device, wherein the display screen is furtherconfigured to display the animation based on a change of the streamingsource device from the second nearby device to the informationprocessing device.
 10. The information processing device according toclaim 1, further comprising: a setting unit configured to enable theinformation processing device to stream the first video to the terminaldevice, based on a user input received from a local user present in thereal space.
 11. The information processing device according to claim 2,wherein the movement control unit is further configured to determine themovement of the information processing device based on the movementcontrol signal received from at least one of a plurality of terminaldevices and based on a determined condition, and the first video isstreamed to the plurality of terminal devices associated with aplurality of remote users.
 12. An information processing method,comprising: in an information processing device including a cameraconfigured to capture real space, a communication unit configured tocommunicate with a terminal device associated with a remote user, and adisplay screen; streaming a first video, captured by the camera, fromthe communication unit to the terminal device based on the informationprocessing device that is selected by the remote user, wherein theinformation processing device is selected from a plurality of devicesthat capture the real space; displaying an image of the remote user onthe display screen based on the first video being streamed to theterminal device; determining a first nearby device of the plurality ofdevices in the real space, wherein the first nearby device captures asecond video streamable to the terminal device; and notifying a presenceof the first nearby device to the terminal device based on adetermination of the first nearby device.
 13. A terminal device,comprising: a communication unit configured to communicate with aplurality of devices, wherein each of the plurality of devices capturesa real space; a streaming control unit configured to playback a firstvideo, that depicts the real space, streamed from a first device of theplurality of devices, wherein the first device is selected by a remoteuser from the plurality of devices, wherein the remote user isassociated with the terminal device; a recognition unit configured todetermine a first nearby device in the real space, wherein the firstnearby device appears in the first video, and wherein the first nearbydevice captures a second video streamable to the terminal device; and adevice control unit configured to: control a display screen of the firstdevice to display an image of the remote user based on the first videobeing played back; and enable the terminal device to change a streamingsource device from the first device to the first nearby device based ona determination of the first nearby device, wherein the streaming sourcedevice streams the first video or the second video to the terminaldevice.
 14. The terminal device according to claim 13, wherein thedevice control unit is further configured to present informationassociated with a functionality of the first nearby device to the remoteuser based on a request to the remote user to change the streamingsource device to the first nearby device.
 15. The terminal deviceaccording to claim 13, wherein the recognition unit is furtherconfigured to determine the first nearby device based on a detection ofa marker, wherein the marker is displayed by the first nearby device inthe first video.
 16. The terminal device according to claim 13, whereinthe plurality of devices includes at least one mobile device, whereinthe at least one mobile device includes a movement mechanism, and thedevice control unit is further configured to generate a movement controlsignal that is transmitted to the at least one mobile device, whereinthe movement control signal controls a movement of the at least onemobile device based on a user input from the remote user.
 17. Aninformation processing method, comprising: in a terminal device thatincludes a communication unit configured to communicate with a pluralityof devices, wherein each of the plurality of devices captures a realspace; playing back a first video depicting the real space, wherein thefirst video is streamed from a first device of the plurality of devices,wherein the first device is selected by a remote user from the pluralityof devices, wherein the remote user is associated with the terminaldevice; determining a first nearby device in the real space, wherein thefirst nearby device appears in the first video, and wherein the firstnearby device captures a second video streamable to the terminal device;controlling a display screen of the first device to display an image ofthe remote user based on the first video being played back; and enablingthe terminal device to change a streaming source device from the firstdevice to the first nearby device based on a determination of the firstnearby device, wherein the streaming source device streams the firstvideo or the second video to the terminal device.
 18. The informationprocessing device according to claim 1, wherein the communication unitis further configured to receive a streaming request and image data ofthe remote user, wherein the streaming request is based on theinformation processing device that is selected by the remote user, andwherein the streaming control unit is further configured to stream thefirst video to the terminal device based on the streaming request.